Comparing Classification Methods for Campaign Management: A Comparison of Logistic Regression, k-Nearest Neighbour, and Decision Tree Induction

Extensive research has been performed to develop appropriate machine learning techniques for different data mining problems, and has led to a proliferation of different learning algorithms. However, previous work has shown that no learner is generally better than another learner. For repeated data mining tasks practitioners have started conducting empirical comparison studies to determine the best technique. Comparing different machine learning methods, however, is a non-trivial task and depends very much on the characteristics of a particular data set and the requirements of the respective business domain. Data mining literature provides little guidance in this respect. Selecting customers which are likely to respond to direct marketing campaigns is an important task in marketing departments, and one where machine learning techniques have been used repeatedly. A systematic comparison of classifier performance can achieve considerable gains in marketing effectiveness. This case study provides an assessment of the predictive performance of different classification methods for SMS campaign management in the telecommunications industry. The analysis is based on a large data set with 165 variables and 10,000 instances. The evaluation of data mining methods for marketing campaigns has special requirements, compared to other application domains. Whereas, typically the overall performance or the error rate are important selection criteria, for campaign management it is more important to select the technique which performs best on the first few quantiles of the overall customer base. In addition, one typically has to deal with very large data sets. This study selects candidate techniques and relevant evaluation criteria for campaign management and provides a guideline for similar comparison studies.